Method for a group of services to operate in two modes simultaneously

ABSTRACT

The invention is a method of handling groups of services where the makeup and structure of the groups can be determined and changed while the application is running. In one embodiment a group of services is grouped as a coordinator cohort group with respect to one client and, at the same time, as a peer group with respect to another client. This is accomplished by registering with a lookup service a coordinator cohort group proxy and a peer group proxy. Clients may download from the lookup service either group proxy in order to use the group of services in the desired mode.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional application of U.S. application Ser.No. 09/940,367 filed Aug. 28, 2001, incorporated herein by reference.

BACKGROUND OF INVENTION

A distributed system is a collection of autonomous computing entities,hardware or software, connected by some communication medium. Whileoften the computing entities are geographically dispersed, in someinstances they might be separate processors in a multi-processorcomputer or even separate software routines executing in logicallyisolated memory space on the same computer. A computing entity need notbe a traditional computer, but more generally can be any computingdevice, ranging from a large mainframe to a refrigerator or a cellphone. A distributed application is an application that executes on adistributed system and one in which parts of the application execute ondistinct autonomous computing entities.

Whenever a distinct component of a distributed application requestssomething (e.g., a data value, a computation) of another component, theformer is called a client and the latter is called a service. It isworth noting that the terms service and client are not exclusionary inthat an item can be both a client and a service. For example, a routinethat calculates the time between two events may be a client and of aclock service; if the clock service then calls a routine that convertsto Daylight Savings Time, the clock becomes a client and the DaylightSavings Time converter is its service.

FIG. 1 shows a typical distributed application of the existing art.There are two clients 2, 4 and four services 10, 12, 14, 16 that theclients 2, 4 might need. Each service has a service proxy 10 a, 12 a, 14a, 16 a which is a module of mobile code that can be used by clients toinvoke that service. A service proxy 10 a, 12 a, 14 a, 16 a contains thecode needed by a client 2, 4 to interact with a service. For instance ifa service is a digital camera on a robotic arm, the interfaces mightinclude Initialize( ), Zoom( ), Rotate( ) and Get_Picture( ). Theservice proxy 10 a, 12 a, 14 a, 16 a may also provide the expectedreturn values for the service, which might include error codes as well.

Mobile code generally refers to a computer program that can be writtenon one platform and executed on numerous others, irrespective ofdifferences in hardware, operating system, file system, and many otherdetails of the execution environment. In addition to independence fromthe physical characteristics of the execution environment, a mobileprogram may move from one computer to another in the middle of itsexecution.

Mobile code may be pre-compiled, or compiled when it arrives at theexecution platform. In the first case, numerous versions of the programmust be written and compiled, then matched across run-time environments;this is mobile code in the letter, but not the spirit, of thedefinition. In addition, the same pre-compiled program cannot move fromone platform to a different one during its execution. In the second, theprogram text may be distributed along with configuration scriptsdescribing what to do in each execution environment. This distributesand delays the specificity of the pre-compiled option. The moreinteresting, and far more common approach exploits a standard virtualmachine, which finesses all the issues of platform heterogeneity. Thevirtual machine is a program that itself mitigates the machinedependencies and idiosyncrasies, taking the raw program text andcompiling it into binary executable.

In addition to clients 2, 4 and general services 10, 12, 14, 16, alldistributed applications need some mechanism for clients 2, 4 to findservices. Often such knowledge is assumed a priori, but many distributedapplications use a look-up service 20. The look-up service 20 is aservice with which the other services are registered or advertised to beavailable to for use by clients. In a simple system, where there is noattempt to coordinate replicas of services, each new service registerswith the look-up service 20 (in the case of replicas, the onus falls onthe client to resolve conflicts and ambiguity). When a service 10, 12,14, 16 registers, it provides information telling clients 2, 4 how tofind it. Commonly, this is a physical location such as an IP address andport number, but in the most modern systems this can be as powerful asgiving the look-up service 20 a service proxy 10 a, 12 a, 14 a, 16 a,which is actual mobile code that clients 2, 4 can execute and use toinvoke that service 10, 12, 14, 16. In this way, the service proxy 10 a,12 a, 14 a, 16 a contains not just location information, but informationfor how to use the service 10, 12, 14, 16. While just as necessary forthe client 2, 4 as location information, this has previously beenassumed as a priori knowledge. When a client 2, 4 wishes to work with aservice 10, 12, 14, 16 it finds it through the look-up service 20,downloads the service proxy 10 a, 12 a, 14 a, 16 a for that service 10,12, 14, 16 from the look-up service 20, then uses the service proxy 10a, 12 a, 14 a, 16 a to invoke the service 10,12,14,16. The look-upservice 20 may also have attributes of the services 10, 12, 14, 16, suchas whether it is a grouped service, what type of group it is, what itscost to use is, how accurate it is, how reliable it is, or how long ittakes to execute. In such cases the clients 2, 4 can use the attributesto decide which of a number of services 10, 12, 14, 16 it wishes to use.

Each of the foregoing has access to a communication network 22 so thatit is capable of communicating with at least some of the other membersin the distributed computing application. The communication network 22may be wireless, a local area network, an internal computer bus, a widearea network such as the Internet, a corporate intranet or extranet, avirtual private network, any other communication medium or anycombination of the foregoing.

In the prior art example shown in FIG. 1, one client 2 is a trafficmonitoring program that notifies a user when and where traffic hasoccurred and the other client 4 is an automated toll collection program.The services are a clock 10, a road sensor 12 that monitors traffic flowon a highway, a toll booth sensor 14 that detects an ID device in eachcar that passes through the toll, and a credit card charge program 16.When each service 10, 12, 14, 16 becomes available to the application itregisters with the look-up service 20 and provides the look-up servicewith its service proxy 10 a, 12 a, 14 a, 16 a.

When the traffic monitoring client 2 begins, it queries the look-upservice to see if a clock is available and what sensors are available.The look-up service 20 responds by providing the client 2 with the clockproxy 10 a, the road sensor proxy 12 a and the toll booth sensor proxy14 a. The traffic monitoring client 2 uses the service proxies 10 a, 12a, 14 a to invoke the clock 10 and the sensors 12, 14, and then tomonitor traffic at various times of the day.

Similarly when the toll collector client 4 begins, it queries thelook-up service 20 to see if a toll booth sensor 14 and a credit cardcharge service 16 are available. The look-up service 20 responds byproviding the client 4 with the toll booth sensor proxy 14 a and thecredit card charge proxy 16 a. The toll collector client 4 uses theservice proxies 14 a, 16 a, to invoke the toll booth sensor 14 and thecredit card charge program 16, and then to identify cars that passthrough the toll booth and charge their credit cards for the toll.

A known feature of distributed applications is that services may begrouped. For instance there may be several services capable ofperforming the traffic sensor functionality. These can be grouped toform a logical notion of traffic sensor that is separate from theparticular implementation of the sensors. This may be done forredundancy purposes in case one of the services fails, to provideparallel processing for computationally intensive tasks, to provideextra capacity for peak loads, as well as for many other reasons.Services in a group may communicate with each other to coordinate theiractivities and states. For instance in the example shown in FIG. 1 itmay be advantageous to group the two sensors 12, 14.

There are two primary types of group structures: the coordinator cohort(CC) group and the peer group. In a CC group there is one distinguishedmember of the group, the coordinator, that processes requests fromclients. The coordinator periodically updates the other services in thegroup, the cohorts, with information about its current state andcompleted requests, so that if the coordinator fails, the cohortselected to replace it will be as current as possible. The more frequentthe updates, the more tightly coupled the states are between groupmembers, and so the more likely the transition will occur without beingvisible to existing clients of the group. On the other hand, morefrequent updates require additional computational capacity andcommunication bandwidth.

In a peer group, all of the members of the group process requests from aclient, which itself requires some logic to decide how to use themultiple results returned from the group members. For example, if threethermometers exist in peer group, and a client requests the temperatureit will receive three answers. Many options exist for using the multipleresults, such as taking the first to respond, taking the average valueof all the responses, or taking the highest value. A peer group is morerobust and fault-tolerant than a CC group because each of the groupmembers should always be in the correct state, and because thelikelihood of the representative member (which is all members in a peergroup, but only the coordinator in a CC group) being unavailable isdrastically lower. However, a peer group also requires more resources,both bandwidth and computational, than a CC group because all of thegroup members are working and responding to each client request.

Another technique known in the existing art is leasing. A lease is animportant concept throughout distributed computing, generally usedbetween a client and service as a way for the service to indicate itsavailability to the client for a length of time. At the end of thelease, if the lease is not renewed, there is no guarantee ofavailability. In a simple example, a service may register with a look-upservice and be granted a lease for five minutes. This means that thelook-up service will make itself available to the service (i.e., listit) for five minutes. If a camera grants a lease to a client for twominutes, then that client will be able to position, zoom, and takepictures for two minutes. There are a wide variety of ways to handlelease negotiation, renewal and termination which are well known to thoseskilled in the art of distributed computing and all such methods aremeant to be incorporated within the scope of the disclosed invention. Adetailed explanation of leases can be found in, Jim Waldo, The JiniSpecification, 2nd Edition, chapter LE (2001), which is incorporatedherein by reference.

One useful aspect of leases is that they can be used for simple failuredetection. If the expectation is that a client will continue to requestlease renewal from a service, but then does not renew its lease, theservice may assume that the client has failed, or is otherwiseunavailable. This allows the service to more efficiently manage its ownresources, by releasing any that were dedicated to expired clients. Sucha use of leasing is described in U.S. Pat. No. 5,832,529 to Wollrath etal.

This is especially important because components only rarely plan andannounce their failure and are not able to predict network outages. Itis far more common that failures and outages are unexpected, and thatthe consequence is an inability to announce anything. In these cases, aclient will not renew its lease so that eventually, the granting servicewill reallocate its resources. The shorter the lease period, the soonera failure can be detected. The tradeoff is that both client and servicespend proportionately more time and resources dealing with leasing andthat timing anomalies may have implications for correctness.

Some benefits of distributed computing and mobile code can immediatelybe seen from this example. First, the clients 2, 4 in FIG. 1 do not needto know ahead of time which sensors 12, 14 are available, or even howmany. They simply query the look-up service 20, which provides thisinformation along with the necessary mobile code 12 a, 14 a to call thesensors. Similarly, the clients 2, 4 do not care which clock 10 isavailable, as long as any clock 10 is available. Again, this is becausethrough the use of mobile code, a client 2, 4 is provided with thenecessary service proxy 10 a to invoke and work with the clock 10. Also,the failure or unavailability of a single sensor 12, 14 or other serviceis not likely to cause the entire application to stop running. Further,the processing load is distributed among a number of computing devices.Also, the various computing entities need not use the same operatingsystem, so long as they conform to a common interface standard.

Jini is one example of a commercially available specification for adistributed object infrastructure (or middleware) for more easilywriting, executing and managing object-oriented distributedapplications. Jini was developed by Sun Microsystems and is based on theJava programming language; consequently, objects in a Jini system aremobile. Jini is described in Jim Waldo, The Jini Specification, 2ndEdition (2001). The Common Object Request Broker Architecture (CORBA),developed by the Object Management Group, and Distributed ComponentObject Module (DCOM), developed Microsoft Corporation, are two othercommercially available examples that are well known in the prior art.Jini, DCOM, CORBA and a number of other distributed computingspecifications are described by Benchiao Jai et al., Effortless SoftwareInteroperability with Jini Connection Technology, Bell Labs TechnicalJournal, April-June 2000, pp. 88-101, which is hereby incorporated byreference.

Distributed computing systems with groups can also be found in the priorart, particularly in the academic literature. For example, OzalpBabaoglu et al., Partitionable Group Membership: Specification andAlgorithms, University of Bologna, Department of Computer Science,Technical Report UBLCS-97-1 describe groups, but assumes the services inthe group are group-aware. Similarly static group proxies, or softwarewrappers, for clients have been described in Alberto Montresor et al.,Enhancing Jini with Group Communication, University of Bologna,Department of Computer Science, Technical Report UBLCS-2000-16, butthese group proxies cannot be modified during execution of thedistributed application to accommodate changes in group make-up andstructure.

A number of problems can be found in these and other implementations andputative descriptions of distributed applications. Chief among these isthat, even if some notion of groups is available within theinfrastructure, both services and clients need to be group-aware; thatis they need to contain logic to interact either within and as part of agroup (in the case of grouped services), or with a group (in the case ofclients of a group of services). This logic is very complex and theskill set required to write such software is very different from theskills required to write the underlying client or service. Further, manyexisting clients and services exist that do not have group logic, andeven for clients and services that are being newly written it can bechallenging to write this logic as part of the module. Even if grouplogic is coded into new clients or services, they become locked into aparticular instance and type of group and in most cases will need to berewritten if the group architecture or makeup changes. Therefore it isdesirable to develop a methodology wherein the group-aware logic forclients and services are provided in separate code modules. Existing andpreviously described attempts at group services have always assumed thatboth the services to be grouped and the clients using group services aregroup-aware. The assumption of group-awareness prevents existing, orlegacy, software from being able to take advantage of the benefits ofgroups (unless they are rewritten) and burdens new applications withproviding the necessary group logic to operate with the particularimplementation of the group service. If wrappers were considered forgrouping legacy services, they were static and hard-coded, locking theservice into a single framework. Moreover, static wrappers introduce anadditional, distinct point in the computation, with negative performanceand, ironically, fault tolerance implications, since such solutions cannever operate in the same process space. In all frameworks, groupstructures were static and therefore did not permit transitions betweengroup structures.

All previous frameworks also ignored clients. Further, even if clientsare written to be group-aware, they must be group-aware in the veryparticular way that the group of services are implemented. For example,if a client is capable of delaying its requests during membershipchanges to a group of services, until it receives a signal informing itthat the membership change has completed, then it cannot interact with asystem in which groups send no such signal, but instead expect theclient to poll for this information. Therefore it would be preferablefor this logic to be provided at run time when the groups areestablished.

A major problem with current distributed computing methodologies thatsupport groups is that changes to the group's membership are invasive;that is, services within a group cannot be changed without temporarilyhalting the availability of the application. Further, in currentsystems, if a service, whether grouped or single, is unavailable, theclient is burdened with handling this unavailability; if it does not,the client may wait indefinitely, take incorrect steps, or even crash.This is true, even in simple redundant backup systems, where the clientmust handle any delays caused by the switch from a primary to a backupservice. Another limitation of current approaches is that groupstructure, CC, peer or otherwise, is not modifiable without alsostopping and then restarting the application, again leaving existingclients in the lurch. Fluid group structure transitions could be used toincrease or decrease quality of service properties such asload-balancing or fault tolerance, and to simplify peer group operationwhen the service code calls for external interaction.

Thus, it is desirable to have a distributed application in which newservices can be added, or services in a group restructured, “on thefly”, that is without halting other members of the application.

It is therefore an object of this invention to provide a method fortransparently managing and interacting with groups of services in adistributed application in which groups are dynamic in their membership,organizational structure, and their members' individual functionality.

It is a further object of this invention to provide a method of handlingtransitions in a group of services that does not burden the client.

It is a further object of this invention to provide a method forgrouping services wherein a group of services can simultaneously bearranged in multiple group modes.

It is a further object of this invention to provide a method of groupingservices in which the group-aware logic is provided in separate codemodules from the core functional logic of the clients and services.

It is a further object of this invention to provide a method of groupingservices in which the code modules that handle the group-aware logic arehighly reusable from one application to the next.

It is a further object of the invention to provide for a method ofgrouping services where services can be added or removed, and groupsrestructured during operation, yet without interrupting, execution.

BRIEF DESCRIPTION OF THE INVENTION

The present invention is a method of handling a wide range of dynamicgroups of services where the makeup of the groups can be determined andchanged while the application is running. This is mainly accomplishedthrough a group proxy, which is generated at run time, and which handlesinteractions with groups of services on behalf of one or more clients.The group proxy consists of a group logic shell which contains all thegroup-aware logic, and a service proxy for each service in the groupwhich contains the necessary logic to interact with the particularservice. The group proxy, which is given to a client for all of itsinteractions with the group of services, buffers calls from that clientto its group when the group is unavailable because it is in transition.When the transition is complete the group proxy transmits the storedclient commands to the group. In the preferred embodiment of theinvention, all the group-aware logic for a distributed computingapplication is provided in separate code modules, namely the groupproxy, group service and grouping agent, thus relieving clients andservices from providing this logic and maintaining the purity of thelook-up service and other infrastructure services.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a distributed computing application of theprior art.

FIG. 2 shows an example of an improved distributed computing applicationof the current invention.

FIG. 3 shows a generic representation of the current invention.

FIG. 4 shows a Foo service joining as the k^(th) member of a coordinatorcohort group.

FIG. 5 shows a Foo service joining as the k^(th) member of a peer groupof Foos.

FIG. 6 shows a fail-over from Foo-1 to Foo-2 in a coordinator cohortgroup

FIG. 7 shows a peer group of services reorganizing itself as acoordinator cohort group to call another service.

FIG. 8 shows a peer group reorganizing itself as a coordinator cohortgroup for use by clients.

FIG. 9 shows an example of a group being in both peer and CC modes.

DETAILED DESCRIPTION OF THE INVENTION

This invention is related to the invention described in the previouslyfiled U.S. patent application, Ser. No. 09/928,028, Group Proxy andMethod for Grouping Services in a Distributed Computing Application,filed on Aug. 10, 2001, which is hereby incorporated by reference.

FIG. 2 shows an example of a distributed computing application of thecurrent invention. As in FIG. 1 there is a communication network 22, alook-up service 20, a number of clients 2, 4, and a number of services10, 12, 14, 16, 18, each of the latter having a service proxy 10 a, 12a, 14 a, 16 a, 18 a. In the current invention some of the services aregrouped. In this example one group of services is a CC group 50 and theother group is a peer group 52. To support the group activity eachgrouped service is provided with a grouping agent 10 b, 12 b, 14 b, 16b, 18 b and there is a group service 24. In addition to there beingproxies for each service there are also group proxies 40, 42, which actas proxies for each group.

The example shown in FIG. 2 provides specific clients services andgroups, but the invention is generic in application and the example isnot meant to limit the invention in any way.

Overview

Prior to describing the method for handling transitions it is helpful tounderstand the system on which the method is executed. While thedetailed workings of the present embodiment of the invention will bedescribed below, a general introduction is provided here using theexample shown in FIG. 2. As in FIG. 1, the example of FIG. 2 is relatedto traffic monitoring and toll collection. An additional service, a logservice 18, has been added which copies all information sent to it tosome form of non-volatile memory. The log service 18 is essentially arecorder. The non-volatile memory might be a magnetic or optical medium,or even a paper print-out.

In this embodiment of the invention the road sensor 12 and the tollbooth sensor 14 are grouped together in a CC group 50. As in FIG. 1 thetraffic monitor client 2 makes calls to a clock 10, which is notgrouped, and a sensor. However, in this example the sensor is grouped.From the point of view of the traffic monitor client 2, it does not needto know that the sensor is grouped, it simply calls a sensor service toget road traffic information, which in this case is a CC group 50. Inthe example the road sensor 12 is the coordinator and the toll boothsensor 14 is the cohort. If the road sensor 12 becomes unavailable, dueto failure or any other reason, the toll booth sensor 14 will act as itsbackup and become the coordinator. The road sensor 12 might bedesignated as coordinator simply because it was the first to registerwith the group service 24, is more accurate, is more reliable, is lessexpensive or for any other reason.

The credit card charge service 16 and log service 18 are also groupedtogether, in this case as a peer group 52. Because they are grouped as apeer group, calls by any client to the credit group service 52 areexecuted by both the credit card charge service 16 and the log service18. This is convenient in that a permanent record of charges is made bythe log service 18 so that audits can be made to make sure that allcredit charges executed by the credit charge service 16 were properlycredited. In the event the credit card charge service 16 becomesunavailable, instead of failing, the credit group service 52, throughthe log service 18, at least creates a permanent record of charges,which can be retrieved later and processed.

Grouping Agent and Group Service

An improvement of the current invention is the use of grouping agents 12b, 14 b, 16 b, 18 b, to handle the group-aware logic for the groupedservices 12, 14, 16, 18. It is the grouping agent that intercept aregistration call from a service to the look-up service 20 and directsthe call to the group service 24. It is also the grouping agents 12 b,14 b, 16 b, 18 b, that handles coordination between the services in agroup. If a service belongs to more than one group, it might havemultiple grouping agents.

While in a new service being written from scratch the grouping functionsperformed by the grouping agent can be written as an integrated part ofthe service, it is preferable that the grouping agent be written as adistinct code module from the core functions (i.e., addition andsubtraction in a calculator). This allows 1) the grouping agent to bemodified without affecting the core, 2) the core to operate withnumerous different (or no) grouping agents simultaneously, 3) thegrouping agent code to be used with a variety of different services, inmost cases, with only minor modification, and 4) grouping agents to beswitched on the fly. In services that are not group-aware, a groupingagent can be added to the existing core to make the legacy servicegroup-aware.

The invention further provides for a novel group service 24 whichperforms a variety of functions that facilitate groups in theapplication. All of the services that wish to be grouped register theirservice proxies with the grouping service 24 instead of the look-upservice 20. More accurately, a service's grouping agent registers itsservice proxy with the grouping service. However, for purposes ofsimplicity any group related activity described as taken by a serviceshall mean that the action is taken either by the service itself, if itis inherently group-aware, or by its grouping agent. The group service24 then registers the appropriate service proxies with the lookupservice 20. The group service 24 also coordinates whether each groupwill be a CC or peer group, and each such group's operation,transitions, and interactions. Most importantly the group service 24dynamically creates the group proxies 40,42 for each group by adding theappropriate service proxy (in the case of a CC group) or proxies (in thecase of peer group) 10 a, 12 a, 14 a, 16 a, 18 a to the appropriategroup logic shell 30, 32, and then the group service 24 registers thegroup proxies 40, 42 with the look-up service 20 for use by the clients2, 4. The group service 24 also coordinates the activities of the groupproxies 40, 42 during fail-overs or other transitions and handles theupdating of group proxies 40, 42 with the look-up service 20 and thevarious fielded (i.e. already attached to a client) group proxies 40, 42when it is necessary to add, delete or switch the service proxies 10 a,12 a, 14, 16 a, 18 a. The group service 24 also handles the swapping ofgroup proxies 40, 42 if a group switches from CC mode to peer mode orvice versa.

Group Proxy

The group proxy 40,42 represents another improvement of the currentinvention. Its task, as each grouping agent does for its service, is tohandle all the group-aware logic for its client. It can be thought of asa device driver for a group of services. In addition, and of particularimportance, a group proxy can buffer or redirect communication to andfrom a client when the group that client is calling is in transition.Such a transition may occur due to a failure of a service in a group,the addition or removal of a service in a group, changing ofcoordinators in a CC group, or a group switching between CC and peermode. Since the group proxy provides an easily configurable softwarelayer between the client and the rest of the distributed application itcan also be used to perform other useful tasks such as copying commandsto a test service, resolving the results of multiple responses from apeer group of services, or copying communication to a log service.

The group proxy 40,42 is made up of a group logic shell 30, 32 and oneor more service proxies 10 a, 12 a, 14 a, 16 a, 18 a. The group logicshell 30, 32 contains all of the necessary group logic for a client tointeract with a group of services. Assuming there is a defined interface(e.g. syntax) to call a service, the group logic shell 30, 32 containsthis interface to present to clients 2, 4. The group logic shell 30, 32contains the logic to intercept client 2, 4 commands to a group 50, 52,store them, and retransmit the commands at a later time. The group logicshell 30, 32 may also contain logic to copy or redirect client 2, 4communication to other services. However, the group logic shell 30, 32does not contain the necessary mechanisms, such as wire protocolimplementation, to interact with the services 10, 12, 14, 16, 18 withina group. These are contained within the service proxies 10 a, 12 a, 14a, 16 a, 18 a. The group service 24 bundles the group logic shell 30, 32with one or more service proxies 10 a, 12 a, 14 a, 16 a, 18 a to form agroup proxy 40, 42.

As shown in FIG. 2, there are separate group logic shells for a CC group30 and for peer group 32. In fact, in the current embodiment there aretwo group logic shells for each group, one peer and one CC. Although alarge portion of the group logic shell code is the same from group togroup, each group has its own shells because the group logic shell hasto present the identical interface to the client as any single member ofthe group would present. In an alternative embodiment, the group logicshells 30, 32 for each group stored within the group service 24 areidentical, and when a group logic shell initializes it receives thenecessary service interface from the grouping agents, or determines theappropriate interface using a process known as reflection. Reflection iswell known to those skilled in the art of object-oriented computing andprogramming languages, and will not be elaborated upon here. Sincestorage space is generally inexpensive and the executable code for thegroup logic shells is not unduly large, in the shown embodiment thegroup service 24 stores a set of two group shells, peer 32 and CC 30,for each group.

In an alternative embodiment, the peer and CC group logic shells 32, 30are combined into a single mobile code module and the group service 24simply tells the group proxy in which mode to act. Such an architecturehas certain advantages when it is desirable to transition groups betweenCC and peer mode during, yet without interrupting, execution, since itis not necessary to switch group proxies or logic shells at the clients,and therefore it is easier to ensure that no client commands are droppedin transition.

The use of a group logic shell to form a group proxy is an improvementof the current invention. It makes it possible to create and reconfiguregroup proxies on the fly as the application is running. It enables anarchitecture where, in most cases, only service proxies in the groupproxy need to be updated as services are added and deleted from a group,instead of replacing the entire group proxy. Alternatively, logic shellsmay be changed, perhaps to switch between peer and CC modes, withoutreplacing the service proxies.

FIG. 2 demonstrates another improvement of the current invention, namelythat the same service can be simultaneously grouped and ungrouped withrespect to different clients. In FIG. 2 the traffic monitor client 2calls the sensor group 50 which includes the toll booth sensor 14.Simultaneously, the toll both sensor 14 is called directly by the tollcollector client 4. The difference is that the toll collector client 4uses the toll booth sensor service proxy 14 a directly, while thetraffic monitor client 2 uses the sensor group proxy 40. As shown theroad sensor 12 is the coordinator of the sensor group 50 so that thesensor group proxy 40 attached to the traffic monitor client 2 isbundled with the road sensor service proxy 12 a. Although not shown, ifthe toll booth sensor 14 becomes the coordinator for the sensor group50, the group service 24 would swap the toll booth sensor service proxy14 a for the road sensor service proxy 12 a in the sensor group proxy 40at the traffic monitor client 2. Then both clients 2, 4 could use thetoll both sensor 14 simultaneously, assuming it had enough processingpower and bandwidth to serve both. Such a configuration may require amore sophisticated grouping agent that is able to differentiate betweencalls to the group and calls directly to the service. In such a scenarioit is also beneficial that the client querying the look-up service beable to establish whether a particular service is grouped or ungrouped.

The group service manages the membership and structure of groups ofservices, is responsible for registering each group with the look-upservice when its composition and structure are stable, andde-registering it when these are in transition. By way of an example, ifthere are three distinct services that have indicated (possibly througha grouping agent) a desire to form a particular group, the group servicemight determine that the instance with oldest time stamp be therepresentative provided to the look-up service; upon monitoring thatinstance the group service might later determine that some otherinstance (e.g., with the next oldest time stamp) should replace it andbe registered with the look-up service. The group service also providesgroup proxies and is responsible for alerting clients through the groupproxies of transitions within a group. The group service may alsodetermine into which group structure the services are organized.

In the present embodiment of the invention it assumed that all groupmembers expose and implement the same external interface. This makes allservices in a group appear to be identical, even if they are not exactreplicas. For example, a group of calculators may each perform addition,subtraction, multiplication and division. Regardless of whether thecalculators were true identical replicas, as long as they implement thesame interface they can easily be grouped in CC or peer group modes. Inthe likely event the actual programmer interfaces are not identical, asingle interface must be decided on by the system architect, and theservice proxy can implement the interface and its translation to theactual programmer interface. Consider that the Calculator group desiresto provide a multiplication function, and consider that Calc-1 nativelyprovides the interface Mult (float x, float y) and returns the result ofx multiplied by y, while Calc-2 provides the interface multiply_by(floatx, float y, float z) and returns the result of x multiplied by y in thevariable z. The system architect may decide that the Calculatorinterface will have syntax Multiply(float x, float y) and provide theresult of x multiplied by y. Then the service proxy for Calc-1 willimplement Multiply(x, y) as Mult(x, y), while the service proxy forCalc-2 will implement Multiply (x, y) as multiply_by (x, y, z), havingpreviously declared its own local variable z, and then return the valuez. To further the example, suppose Calc-3 supports 64-bit precision, butMultiply (x, y) provides for only 16-bit precision; then the serviceproxy for Calc-3 will need to truncate 48 bits. If a member of the groupcannot perform all the functions defined in the common interface, thenthe service proxy will need to compensate, either by completing thefunctionality, or by returning an exception (provided exceptions aredefined in the common interface). For instance, suppose Calc-4 providesonly for addition. Then its service proxy could implement Multiply (x,y) as y additions of x to itself (for example: float result=0.0; for inti=1 to y, {result=add(x, result)}).

While in the preferred embodiment, the translations necessary to providea common interface are handled by the service proxies, a similarfunction can be performed by the grouping agent for the service. Takingadvantage of mobile code, another solution to this problem is to providea special dedicated wrapper to the client or the service to handle thistranslation. Other solutions will be obvious to those skilled in theart, and are included within the scope of this invention. In analternative embodiment services that do not present the same interfaceare grouped together.

The invention is not meant to be limited to the particular applicationor number of services, groups and clients shown in FIG. 2. FIG. 3 showsa generic implementation of the present invention in which there arethree clients 2, 4, 6 and three different groups of services 50, 52, 54,although there need not always be an equal number of clients and groups.In this representation groups are represented in capital letters andservices in small letters. For each group 50, 52, 54 the group service24 has a CC group logic shell 30, 34, 38 (indicated by a subscript “c”)and a peer group logic shell 32, 36, 39 (indicated by a subscript “p”).One point of this representation is to demonstrate that a client cancall multiple groups, and a single group can be called by multipleclients, provided that each client 2, 4, 6 has the appropriate groupproxy 40, 42, 44. For instance one client 2 calls all three groups: A50, B 52, and C 54. Likewise, one group, C 54, is used by all threeclients 2, 4, 6, and therefore each client has the group proxy 44 forthat group. Also, in this representation there is a group, group A 50,consisting of only one service, thereby allowing the client of a singleservice to obtain some of the benefits of the group proxy, such asfailure masking by buffering. In this embodiment, as presently shown,group A 50 and group B 52 are peer groups, and group C 54 is a CC group,although the structure of each group can be reconfigured.

While this description has principally referred to two types of groupmodes, peer and coordinator cohort, hybrids of these types, and othertypes of modes are possible, and the invention is meant to incorporateall such group modes, whether currently existing or invented hereafter.It has also been assumed herein that a grouping agent contains all thenecessary logic to act in either CC or peer mode. However in analternative embodiment, a service may have separate grouping agents forCC and peer modes. Likewise, although not optimal for reasons discussedabove, a service could be written to incorporate the grouping agentfunctions, without having a separate group agent.

A group service is not necessary to gain the client-side benefits ofcommand buffering using a group proxy. As described, the group serviceperforms both failure detection and group management. In the absence oftrue groups, but given a mechanism for detecting failures, the “group”proxy could buffer requests upon being notified of a failure. Uponnoticing that the service had been reestablished (for example, byperiodically querying the look-up service) this group proxy would resumenormal operation. This provides for less overall reliability (theexistence of a group of replicas is proportionately more reliable),increased fail-over time (the length of service unavailability due to afailure) and increased latency (the length of time to complete aclient's request), but still shields clients from the effects of servicefailures or transitions. In the preferred embodiment for implementingfault tolerance, the distributed system will implement physicalreplication of services (though they need not be identical replicas),and therefore will have a group service.

It is also possible, in an alternative embodiment, to combine the groupservice and lookup services into a single service. Likewise, in analternative embodiment, the group logic shell, instead of being storedin the group service could be provided by the system designer ahead oftime to each client that will need a particular group, and then thegroup service simply provides and updates the appropriate serviceproxies in those group logic shells. Such an architecture is lessdesirable in that it is less flexible, since it requires prior knowledgefor each client, that it will use a group and which groups a servicewill be using.

The basic methodology for handling transitions within a group is for thegroup proxy to buffer commands while the group is in transition, toupdate or replace the group proxy so that it can work with the revisedgroup, and then for the group proxy to transmit the commands it hasbuffered.

The remainder of the discussion will describe the particular methodologyused to handle fail-overs and other transitions within a group. Bothpeer groups and CC groups are described. In the discussions thatfollows, a generic service will be called a Foo, which could be anyfunctionality. A Foo could be a clock, a counter, a display driver, atraffic sensor, or a calculator. Further a reference to a service takinga particular action being taken by a service shall mean the servicetaking that action either directly, or, in the preferred embodiment,through its grouping agent.

Addition and Removal of Services from a Group

FIG. 4 shows how another instance of a Foo service, Foo-k 14, joins anexisting CC Foo group. In order to join a Foo group, Foo-k 14 (or itsgrouping agent 14 b) queries the look-up service 20 to see if a groupservice is available 401. The group service 24 has already registeredwith the look-up service 20 and has given the look-up service 20 its ownproxy (not shown). The look-up service 20 responds to Foo-k's (or itsgrouping agent's) request by providing it with the group service proxy402. The Foo-k grouping agent 14 b uses the group service proxy toinvoke a method specifying a group name to join (in this case the Foogroup), possibly the group structure it desires to participate in, andprovides the Foo-k service proxy 14 a to the group service 24, 403.Then, since there already is an established coordinator for the Foogroup (assuming it is Foo-1), the group service 24 simply notifies thegrouping agent 10 b for the group coordinator 10 that there is a newmember, or multiple new members, of the Foo group 404. The Foo-1grouping agent 10 b then begins to include the Foo-k grouping agent 14 bin its periodic broadcasts to all the other Foos of its current group405. In an alternate embodiment, the grouping agents would be initiallydesigned to listen for relevant update events, so that updates can bedone without requiring the coordinator to be aware of its cohorts'identities. Analogously, when a cohort Foo service, Foo-j fails or isremoved from the group, in the current embodiment, the coordinator mustbe informed by the group service; in the anonymous embodiment it wouldnot need to be. Removal of a Foo service from the Foo group, other thana coordinator, is similar to adding a Foo service. The Foo-j groupingagent notifies the group service 24 that Foo-j is leaving the service.The group service 24 deletes the Foo-j from the proxy list for the Foogroup, and then instructs the coordinator's grouping agent that it nolonger needs to include Foo-j in its periodic state updates. In theevent that a Foo-j leaving is the coordinator, a new group coordinatormust be designated. This process is similar to the fail-over in a CCgroup described below.

FIG. 5 shows how another instance of a Foo service, Foo-k 14, joins anexisting peer Foo group. In order to join a Foo group, Foo-k 14 (or itsgrouping agent 14 a) queries the look-up service 20 to see if a groupservice 24 is available 501. The group service 24 has already registeredwith the look-up service 20 and has given the look-up service 20 its ownproxy (not shown). The look-up service 20 responds to Foo-k's 14 (or itsgrouping agent's 14 b) request by providing it with the group serviceproxy 502. The Foo-k grouping agent 14 b uses the group service proxy toinvoke a method specifying a group name to join (in this case the Foogroup), possibly the group structure it desires to participate in, andprovides the Foo-k service proxy 14 a to the group service 503.Continuing, the group service 24 deregisters Foo from the look-upservice 20 so that outdated Foo proxies 10 a, 12 a are no longerdistributed 504. The group service adds the Foo-k service proxy 14 a tothe existing set of proxies for Foo members, adding the Foo-k serviceproxy 14 a to the peer Foo group logic shell 32, and re-registers Foowith the look-up service 20, 505. The group service 24 then distributesFoo-k's service proxy 14 a to all fielded peer Foo group proxies (thosealready attached to clients), which add it to the bundle of other Foomember proxies already within the Foo group logic shell 506. Futureclient requests are therefore sent to Foo-k as well as all previous Foogroup members. Steps 505 and 506 can be executed in either order orconcurrently. The group service 24 might also instruct the group proxyfor the clients to buffer commands until they receive the Foo-k proxy42. However, in contrast with a CC group transition, there is generallyno need for group proxies of peer groups to await further informationabout the peer group transition, so that there is no need for peer groupproxies to buffer client commands.

To remove Foo-j from a peer Foo group, the group service 24 distributesinstructions to the Foo peer group proxies 42 (already attached toclients 2) to remove the Foo-j service proxy from each of the Foo peergroup logic shells 32. As in steps 504 and 505 above, the group serviceunregisters then re-registers Foo with the look-up service, and, asabove, the group proxy 42 at the look-up service 20 and clients 2 can beupdated in either order or concurrently.

Fail-overs

FIG. 6 is a description of how the invention handles a fail-over in a CCgroup specifically, and transitions within a group generally. To begin,Foo-1 10 has a lease with the group service 24, where the group service24 is the lease grantor and Foo-1 10 is the lease holder. The groupservice 24 has in turn negotiated a lease for the grouped Foo servicewith the look-up service 20. Foo-1 10 fails and therefore does not renewits lease with the group service 24. The group service 24 assumes thatFoo-1 10 has not renewed its lease because it has failed. The groupservice 24 then cancels the Foo lease with the look-up service 20, 601thereby temporarily preventing any new client from finding the Foogroup. The group service 24 also announces (whether through multicast,broadcast, or individual event notification) to the group proxy 40 usingthe Foo service that Foo is unavailable 602. The announcement may alsobe heard by other interested members of the distributed application,such a log service that records errors or a beeper service that notifiesa human operator. These decisions are left to the system designer, butmay be implemented the same way.

In this example there is a single client 2, but there may be multipleclients using the Foo group, in which case each client would have aninstance of the Foo group proxy 40 and would be notified and updated bythe group service. Likewise, the Foo group proxy 40 for each clientwould buffer that client's commands during any transitions.

While in the described embodiment a service detects a client'sunavailability through leasing, any other method of detectingunavailability can be used. For example, a dedicated failure detectionservice may be employed to actively and interactively monitor the statusof all system components. Many methods for detecting unavailability,whether performed by each service or by a generic failure detectionservice, are known to those skilled in the art, and all such methods, aswell as any others later invented, are included within the scope of thisinvention.

Similarly, while in the described embodiment the group service announcesthe notification of the Foo-1 10 failure, essentially combining thefunctions of failure detection, failure announcement and grouporganization, the system can be designed to separate these functions;specifically, a failure detection service could detect and announcefailures to clients and to the group service, or it could passdetections on to an announcement service.

Continuing in FIG. 6, upon notification of the unavailability of Foo,the group proxy 40 begins to buffer commands to Foo from the client 2 itrepresents. The group service 24 then requests 604 that another Fooservice, in this case Foo-2 12, become the coordinator of the group andsynchronize its state with the remaining Foos 605, 606. The statesynchronization is handled by the grouping agent 12 b, 14 b for each ofthe services 12, 14. Depending on the degree of assurance ofsynchronization required, this can be done anonymously through eventnotification (low degree of assurance) or explicitly throughtightly-coupled individual method invocations (high degree ofassurance). Foo-2 12 becomes the coordinator and then acknowledges thegroup service 24, 607. The group service 24 registers Foo-2 12 as theFoo service with the look-up service 20, 608, preferably by providingthe look-up service 20 with a new Foo group proxy 40 a, 608 a containingthe same group logic shell 30, but now with the Foo-2 service proxy 12a. Alternatively, if the look-up service 20 is capable of modifyingregistered proxies, the group service 24 can provide the look-up service20 with the Foo-2 service proxy 12 a to update the Foo group proxy 40with (but leaving the existing group logic shell 30 in place). The groupservice 24 then distributes the Foo-2 service proxy 12 a to the clients'group proxies (only one shown) 609. The group proxies 40 delete theFoo-1 service proxy 10 a and add the Foo-2 service proxy 12 b, 609 a.The group service 24 then announces (not shown) to all the group proxiesthat the Foo service is again available. Note that steps 608 and 609 canbe executed in either order or concurrently. Using the Foo-2 serviceproxy 12 a the group proxy 40 directs the buffered commands to Foo-2610. Once all buffered command have been sent, the client 2 commands canagain be sent directly.

The handling of a failure of one of the services in a peer group isrelatively trivial. The failure might be detected when a failed Fooservice does not renew its lease with the group service, or when theclient's group proxy detects that a failed Foo did not provide aresponse to an invocation and then notifies the group service 24. Thefailed Foo's service proxy is simply removed from the peer group logicshells at the clients 2 and the look-up service 20 bundle as describedabove with respect to FIG. 8. In a peer group configuration, thetransition period is much short than for a CC group, so buffering maynot be needed.

As in the case of the CC groups, while the details of the peer grouphave been described with a single client, it is equally applicable to anapplication with multiple clients, where each client has a replica ofone or more group proxies. The group service notifies and updates thegroup proxies at each of the clients and each group proxy bufferscommands for the client it is attached to.

Swapping Services

In addition to group membership changes and fail-overs the group proxycan be used to handle other types of transitions. For instance, it maybe desirable to swap one service for another on the fly, that is withoutstopping the application or the without stopping the clients that callthose services. This is easily achieved, for peer or CC groups, with thecurrent architecture. The group service instructs the relevant groupingagents to begin buffering clients' commands, and then deregisters theretiring service's group proxy from the look-up service. The retiringservice's proxy is also removed from the group service. The replacementservice registers with the group service to be a member of the group.The group service then updates the group proxies at the clients with thenew service proxy, or an entirely new group proxy. The group servicethen registers the new group proxy with the lookup service. The groupservice then instructs the group proxies at the clients to again startprocessing commands, and the group proxies release their queue ofbuffered client commands to the replacement service. Alternatively, thereplacement service can be added to the group of the retiring servicefirst, and then the retiring service can be removed. On the fly swappingof services can also easily be accomplished even for a client that isungrouped by treating it as a group of one.

The group proxy can also be used to test a new service that is being runin parallel with an existing service. The service to be tested isregistered as part of the same group as the prime service. Commands fromthe client are transmitted from the group proxy to the prime service aswell as the test service. In a peer group this is accomplished bybroadcasting client commands to all members of the group. In a CC groupit is accomplished by instructing the grouping agent for the coordinator(assuming the prime service is the coordinator) to effectivelysuperimpose a peer group subset containing it and the service to betested (or its grouping agent), though still operating in CC mode withthe remainder of the cohorts. In either peer or CC mode, as responsesare returned to the group proxy it compares results, performance, and soforth, from the service to be tested with the previously establishedservices. Alternately, the group proxy can pass these results to aspecialized benchmarking or evaluation program to perform thesecomparisons, or to a log service for later evaluation. It is alsopossible to have the grouping agent for the test service handle ordiscard the responses. Many other ways of performing comparison will beobvious to those skilled in the art and are incorporated within thescope of this invention.

It is also possible to test an ungrouped service (and even a servicethat is not group-aware) in parallel with a group of services by havingall the group proxies that call the group copy their commands to theservice being tested. Similarly, the group proxies can be used to copyand redirect client communications for other purposes such as recordingcommands to a log service or copying communication to a displaymonitored by a human. The latter use may be particularly useful whendebugging new applications.

Switching Between CC and Peer Modes

As discussed above, many times a service to one client calls yet anotherservice to perform a function, and in such instances the first servicebecomes a client of the second service. For services grouped in peermode, it is often not desirable for the each service in the group tocall the second service. This can result in overloading the secondservice, incorrect answers, delay, or excessive use of bandwidth. Somemechanism is required for the plurality of members of the peer group tosend a single command when they are acting as a client. More generally,there are times when a peer group needs to organize itself internally asCC group, while still appearing to its own clients as a peer group.

The present invention provides a mechanism for accomplishing such aninternal reorganization as shown in FIG. 7. The peer group 50 switchesto a CC group whenever its role switches from service to client; thatis, from provider to requester. In effect, one group member assumes therole of coordinator client, while the remaining group members becomecohort clients. In this case, the re-organization is purely internal tothe peer group; it does not reflect any desire for the group to beperceived externally by its clients as a CC group. In FIG. 7 the Foogroup 50 periodically needs to call another service, named Goo 60. Whenany member of the peer group is ready to issue such a command, in thisexample Foo-2 12, to the Goo service 60 it (or its grouping agent 12 b)notifies the other members 10, 14 of the group 50 (or their groupingagents 10 b, 14 b) 701 which then buffer their own outgoing commands tothe Goo service 60 until told otherwise. The coordinator 12 (or itsgrouping agent 12 b) then makes the relevant invocation on the Gooservice 60, 702. The Goo service 60 processes Foo-2's command andreturns a response 703. When Foo-2 receives the response it distributesit to the cohorts 10, 14. The coordinator indicates that internaloperation of the group can now resume in peer mode. If, during the timewhen the group 50 is in CC mode, it is desirable to prevent any of theFoo services 10, 12, 14 from executing any new commands from clients 4,then grouping agents 10 b, 12 b, 14 b can buffer incoming commands.Alternatively, the Foo service that is acting as coordinator may,instruct the group proxy 42, either directly or preferably through thegroup service 24, to buffer client 4 commands.

In other cases, the transition between group modes, whether peer to CCor CC to peer, may be triggered by an external circumstance. Forexample, going from CC to peer may be occasioned by a policy-based needfor increased fault tolerance, whereas going from peer to CC may beoccasioned by clients accepting a lesser quality of service. An exampleof this might be the traffic monitoring system described in FIG. 2.During normal times it may be sufficient to operate in CC mode with theroad sensor 12 as the coordinator and the toll booth sensor 14 as thecohort, especially if the traffic monitoring client 2 has to pay anextra fee to get data from the toll booth sensor 14. However, duringrush hour the sensor group 50 might switch to peer mode to ensuregreater accuracy of data and provide higher reliability.

In an alternative embodiment, the grouping agents can rely on the groupservice to assist in directing transitions from peer to CC or viceversa, but as long as the reorganization is purely internal the groupservice would not update the fielded group proxies or the group proxybeing distributed by the lookup service.

In cases where the transition is explicit, that is the group not onlyreorganizes its structure, but intends to make itself available toclients in this new structure, the group service would be involved. Inaddition to directing the transitions of the services in the group, thegroup service would coordinate updating the fielded instances of thegroup proxy at clients and the lookup service.

A group mode reorganization from peer to CC mode is shown in FIG. 8 inwhich the change is recognized externally by clients. To begin, thegroup service 24 unregisters the group from the lookup service 20, 801.The group service 24 then creates a new group proxy 40, comprised of theappropriate group logic shell (CC 30 or peer 32) with the appropriateservice proxies 10 a, 12 a, 14 a bundled in it, and distributes the newgroup proxy 40 to the clients 4 to replace the old group proxy 42, 802.When the new group proxy 40 arrives at the client 4 it must handshake803, 804 with the old group proxy 42 in order to be made aware of anyoutstanding requests, or any previously buffered commands fromuncompleted membership transitions. Once this handshake has beencompleted, the new group proxy 40 can take over at that client 4 and theold group proxy 42 can be deleted from that client 4. The group service24 then updates the lookup service 20 with a new group proxy 805, andthen instructs the new group proxies 42 at the clients 4 to begintransmitting commands. While in FIG. 8 the transition is from peer to CCmode, a transition from CC mode to peer is substantially the same

In the described embodiment, because there are different group logicshells for peer and CC modes, the group proxy is completely replacedwhen transitioning between modes. In an alternative embodiment a singlegroup logic shell can contain the logic for both a CC group and a peergroup. With a single group logic shell, in order to switch modes, thegroup service only has to update the service proxies bundled withingroup logic shell at the clients and look-up service. Specifically, in aswitch from peer mode to CC mode, the group service would select oneservice and instruct it (or its grouping agent) to become thecoordinator. The group service would then announce to all fielded groupproxies that are distributed at clients that they should delete all ofthe service proxies except the coordinator's service proxy from thegroup logic shell. In a switch from CC mode to peer mode, the groupservice would add the necessary service proxies to the multipleinstances of the group logic shells at the clients to form a new groupproxy. In both cases, the group service would still unregister andreregister the group with the look-up service to reflect the group's newincarnation. One advantage to this embodiment is that there is no needto handshake and pass buffered commands from one group proxy to another,since the same group proxy remains in place at each client.

In yet another alternative embodiment the group proxy could store allthe service proxies for the group in whichever mode it was operating,and then the group service would not have to add and delete serviceproxies, but simply tell the group proxies at the clients which mode toact in and which proxies to use. In this embodiment service proxieswould still be updated as services were added to and deleted from thegroup.

Dual Group Modes

It is also possible for the same group to simultaneously be present inboth peer and CC modes. The groups may have the identical set of membersor only some members in common. Clients would determine which group modethey desired, for example based on price or quality of service offered,and be provided by the group service with the appropriate group proxy.FIG. 9 shows such a distributed system in which Group B 52 is used byClient 2 4 in peer mode, by Client 3 6 in CC mode, and by Client 1 2 inboth modes. In the example, Service b 12 is the coordinator of Group B52 when it is in CC mode. Each client 2, 4, 6 is provided with theappropriate group proxy 42, 44 so that it can call Group B 52 in thedesired manner. The look-up service 20 has registered a peer group proxy42 and a CC group proxy 44 for Group B 52. It is therefore necessarythat clients that query the look-up service 20 not only be able to finda Group B proxy, but that the lookup service be able to provide clientswith a description of the group proxy's structure (peer or CC).

In the embodiment as shown the grouping agent 12 b, 14 b, 16 b for eachservice 12, 14, 16 detects whether a command is being sent to the group52 as a CC or peer group, and coordinates with the other grouping agentsto appropriately update the states of each of the services. While it maybe desirable for the services to handle multitasking in order toeffectively switch states, this is no different from any service thatcan be called asynchronously by two different clients. Alternatively,the burden of multitasking between various states can be put onto thegrouping agents 12, 14, 16. While in FIG. 9, a single grouping agent 12b, 14 b, 16 b is shown for each service 12, 14, 16, in an alternativeembodiment each service could have a peer grouping agent and a CCgrouping agent.

An interesting aspect of the distributed computing system shown in FIG.9 is that Client 1 2 can call Group B 52 in either CC or peer mode,depending on which group proxy it uses. The client may be executing twoprocesses, one of which uses Group B in peer mode and the other uses itin CC mode. Alternatively, the client may be group-aware and decidebased on certain criteria (reliability, cost, time of day) to executecertain commands in one mode, and certain commands in another mode.

It is understood that the invention is not limited to the disclosedembodiments, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims. Without further elaboration, the foregoingwill so fully illustrate the invention, that others may by current orfuture knowledge, readily adapt the same for use under the variousconditions of service.

1. A computer implemented method of grouping services in a distributedcomputing application comprising the steps of: initiating a plurality ofservices arranged into a group, with each service having its own serviceproxy and grouping agent; each grouping agent registering its associatedservice with a group service including providing the group service withthe service proxy of the service it is registering; the group servicebundling a first group logic shell for a first group mode with at leastone of the service proxies to form a first group proxy for the group;the group service bundling a second group logic shell for a second groupmode with at least one of the service proxies to form a second groupproxy for the group; the group service providing the first group proxyto a first client and the second group proxy to a second client; thefirst client using the first group proxy to call the group in the firstgroup mode; and the second client using the second group proxy to callthe group in the second group mode.
 2. The method of claim 1 wherein thefirst group mode and second group mode are selected from the setcomprising peer and coordinator cohort.
 3. The method of claim 1 whereinthe first client and the second client are the same client.
 4. Themethod of claim 1 wherein the group acts simultaneously in the firstgroup mode and second group mode.
 5. A computer readable mediumcontaining instructions for controlling a computer system to perform amethod of grouping services in a distributed computing applicationcomprising the steps of: initiating a plurality of services arrangedinto a group, with each service having its own service proxy andgrouping agent; each grouping agent registering its associated servicewith a group service including providing the group service with theservice proxy of the service it is registering; the group servicebundling a first group logic shell for a first group mode with at leastone of the service proxies to form a first group proxy for the group;the group service bundling a second group logic shell for a second groupmode with at least one of the service proxies to form a second groupproxy for the group; the group service providing the first group proxyto a first client and the second group proxy to a second client; thefirst client using the first group proxy to call the group in the firstgroup mode; and the second client using the second group proxy to callthe group in the second group mode.
 6. The computer readable medium ofclaim 5 wherein the first group mode and second group mode are selectedfrom the set comprising peer and coordinator cohort.
 7. The computerreadable medium of claim 5 wherein the first client and the secondclient are the same client.
 8. The computer readable medium of claim 5wherein the group acts simultaneously in the first group mode and secondgroup mode.